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In  previous  reports  to  the  U.S.  Army  Research  Institute  we 
have  discussed  simulation  measurement  technology  and  measurement 
reliability.  The  present  report  extends  the  information 
presented  earlier.  Since  our  quasi-exper imental  simulations  were 
specifically  developed  as  assessment  and  training  tools, 
measurement  validity  (for  assessment  purposes)  and  training 
effectiveness  must  be  evaluated.  In  this  paper,  we  will  focus 
upon  assessment  validity.  A  subsequent  report  will  be  concerned 
with  training  effectiveness. 

As  part  of  the  continuing  development  of  the  quasi- 
experimental  simulation  technology,  the  number  of  measures  that 
are  obtained  from  participants  have  continued  to  increase.  With 
as  many  as  sixty  separate  measures  now  available,  it  has  become 
important  to  determine  where  commonalities  among  those  measures 
might  exist.  The  present  report  will  explore  those  commonalities 
via  a  factor  analysis  (varimax  rotation)  of  scores  obtained  by 
more  than  100  individuals  who  participated  in  either  the  Shamba 
or  the  Disaster  simulation  (or  both).  In  addition,  factor 
analysis  allows  the  calculation  of  factor  scores  which  may 
potentially  be  more  appropriate  assessment  indicators  of 
performance  quality. 

In  an  evaluation  of  assessment  validity,  the  present  report 
correlates  performance  scores  and  factor  scores  with  sociographic 
indicators  of  success  (age  relevant  income  and  job  level,  etc.) 
and,  finally,  employs  multiple  stepwise  regression  procedures  to 
select  both  the  theoretical  concepts  and  the  simulation  based 
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measures  that  are  optimal  predictors  of  managerial  achievement. 


6-4 


LOCUS  OF  MEASUREMENT 

We  are  concerned  with  the  measurement  of  managerial 
performance.  Are  simulations,  especially  quasi-exper imental 
techniques,  appropriate  measurement  tools?  Before  considering 
the  specific  characteristics  (and  accomplishments)  of  our 
simulation  based  measurement,  the  uninitiated  reader  may  wish  to 
consider  whether  our  techniques  represent  the  appropriate  locus 
and  level  of  managerial  assessment.  Certainly  one  may  ask 
whether  it  would  not  be  more  reasonable  to  measure  performance  at 
the  organizational  level  (e.g.,  group  productivity,  organi¬ 
zational  output  variables,  cost  and  profit,  group  or  team 
performance,  turnover  and/or  absenteeism).  Certainly  enough 
information  on  turnover  and  absenteeism  has  been  collected  to 
develop  comparative  standards.  Alternatively,  we  might  have 
chosen  to  measure  standard  variables  that  are  concerned  with  the 
manager  as  an  individual,  e.g.,  motivation,  individual  goals,  job 
satisfaction,  role  stress,  and  contributions  to  leadership.  On 
first  thought,  the  choice  seems  a  difficult  one.  However,  is 
there  some  way  to  measure  both  organizational  and  individual 
functioning  at  the  same  time? 

Beyond  questions  about  the  "location"  of  appropriate 
measurement,  one  might  be  concerned  with  the  "kind"  of 
performance  assessment  (and  training).  For  example,  should  we 
apply  judgmental  (i.e.,  subjective)  or  non- judgmental  (i.e. 


objective)  measures  (cf.  Landy  and  Farr,  1983)?  Measurement  of 


output,  of  absenteeism,  profit,  attainment  of  agreed  upon  goals, 
etc.,  would  reflect  objective  non- judgmental  measurement  of 
performance.  Ratings  by  supervisors,  peers  or  even  ratings  of  a 
manager  by  those  supervised  would  represent  subjective 
"judgmental"  approaches.  Ratings  can  be  very  useful:  for 
example,  peer  ratings  during  early  training  can  sometimes  predict 
long  term  success  better  than  other  predictor  variables. 

However,  problems  of  distortion  may  emerge,  especially  at  later 
points  in  a  person's  career.  Political  considerations,  social 
conflicts,  competitiveness,  and  many  other  aspects  of  task, 
organizational  and  interpersonal  components  of  the  work 
environment  can  affect  rating  outcomes.  Leniency,  excessive 
severity,  tendencies  toward  a  common  rating  point  for  all  or  many 
ratees,  halo  effects,  proximity  effects  and  more  have  been 
observed.  Different  effects  of  the  source  of  ratings  are  also 
evident  (Cascio,  1982).  These  differences  are  not  only  due  to 
status  variables,  but  also  an  outcome  of  insufficient  knowledge 
or  inappropriate  orientation.  For  example,  ratings  by  peers  may 
provide  a  valuable  overall  performance  assessment  but  tend  to  be 
insensitive  to  differential  task/job  requirements  (Holzbach, 

1978)  . 

A  developing  body  of  literature  attests  to  the  utility  of 
sel f -appr ai sa 1  techniques  in  which  an  individual  judges  his  or 
her  own  job  performance.  On  the  positive  side,  the  opportunity 
to  participate  in  performance  appraisal,  especiall)  if  combined 
with  goal  setting,  tends  to  improve  an  individual's  motivation. 


On  the  negative  side,  however,  comparisons  with  appraisals  by 
supervisors,  peers,  and  subordinates  suggest  that  self¬ 
appraisals  tend  to  show  more  leniency,  less  variability,  more 
bias,  and  less  agreement  with  the  judgment  of  others.  According 
to  Cascio,  however,  self  ratings  also  tend  to  show  less  central 
tendency.  Cascio  has  concluded  that  se 1 f -a ppr a i sa 1 s  are  more 
appropriate  for  counseling  and  development  than  for  personnel 
decisions  . 

Subjective  e.g.,  rating  measurement  of  performance  is,  in 
other  words,  flawed.  It  may  be  useful  in  some  cases  and  for  some 
purposes.  Some  form  of  objective  measurement  should  be,  where 
feasible,  preferred.  But  objective  performance  assessment  for 
personnel  that  must  deal  with  complex  and  uncertain  tasks  suffers 
from  criterion  problems.  Such  criteria  as  turnover  or  profit  may¬ 
be  affected  by  unrelated  variables  or,  if  not,  are  at  least 
secondary  reflections  of  actual  competence  and  potential.  We 
must  .get  closer  to  the  assessment  of  actual  competence  at  the 
individual  level,  and  to  its  impact  upon  a  variety  of 
organizational  requirements. 

Assessing  and  Training  Managerial  ’’Excellence” 

Despite  the  multiple  approaches  to  assessment  of  performance 
in  complex  tasks,  success  has  been  slow  in  coming.  It  has  been 
especially  slow  where  assessment  of  managers  (executives)  is  the 
focus  of  interest.  The  many  issues  involved  in  the  measurement 
of  executive  performance  have  often  been  debated  and  reviewed. 
Without  question,  the  interest  in  assessing  and  training 


D 

executive  competence,  for  example  executive  task  performance,  has 

been  considerable.  Nonetheless,  only  six  years  ago,  Goldstein 

(1980)  described  research  publications  on  these  management 

functions  as  "low  in  utility"  and  "dominated  by  anecdotal 

presentations."  Goldstein  continues:* 

"Typically,  articles  of  this  type  either  present  a  program 
to  train  managers  to  perform  a  particular  act  such  as 
responding  to  discipline  problems  (e.g.,  Gill  &  Taylor  1976) 
or  they  discuss  a  particular  issue  such  as  the 
identification  of  tomorrow’s  managers  (e.g.,  Muse  1972).  In 
terms  of  empirical  research,  the  exception  to  the  lack  of 
activity  is  found  in  research  on  rater  training.  In 
general,  however,  the  sad  state  of  this  field  is  consistent 
with  the  lack  of  knowledge  and  theory  development  on  such 
topics  as  leadership  and  adult  learning.  McCall  (1976) 
reviewed  this  literature  and  concluded  that  leadership 
training  repeats  the  mistakes  of  leadership  research.  It 
tends  to  focus  on  leadership  style  and  does  not  take  into 
account  the  nature  of  managerial  work  which  often  consists 
of  a  variety  of  fragmented  activities.  Further,  it  only 
includes  situations  limited  to  the  immediate  work  group. 
Stogdill  (1974)  similarly  noted  that  leadership  training  has 
not  examined  the  consequences  of  training  to  determine  the 
effects  on  group  performance." 

Considering  the  lack  of  adequate  success  in  measuring 
executive,  especially  senior  executive  performance,  what  approach 
should  be  optimal?  Ratings,  especially  at  this  level,  are  not 
sufficiently  effective.  Objective  measures  fail  to  point  out 
differences.  As  shown,  senior  executives  all  over  the  world  do 
not  differ  greatly  in  experience,  training,  intellectual 
competence,  and  more.  On  first  view,  then,  we  might  conclude 
that  assessment  of  individual  competence,  for  example  assessment 


*Note  that  the  term  "leadership"  in  this  quote  is  used  in  a 
more  general  sense,  i.e.,  similar  to  executive  quality.  In  other 
words,  it  is  not  restricted  to  the  more  interpersonal  "leading 
people"  orientation. 


of  information  orientation  or  decision  making  performance,  at 
least  at  senior  levels,  might  be  less  than  profitable.  What, 
then,  can  we  do  to  distinguish  between  one  senior  executive  who 
will  function  in  a  mediocre  fashion  and  one  who  would  be 
eminently  successful? 

We  might,  simply,  attempt  to  measure  external  performance 
criteria  that  were  evident  in  that  person's  last  job:  "What  has 
happened  to  an  organization  during  the  tenure  of  that  particular 
executive?"  "How  did  a  military  post  perform  while  a  particular 
officer  was  in  charge?"  -  How  well  would  such  an  approach  work? 

Is  the  performance  of  an  organization  or  of  an  organizational 
unit  a  true  reflection  of  executive  competence?  At  times  it  is 
not.  One  shortcoming  of  basing  assessment  upon  such  external 
criteria  has  been  suggested  by  Guion  (1965).  The  externally 
defined  success  of  an  executive  lies  largely  in  meeting  major 
organization  goals  through  the  coordinated  efforts  of  the 
organization.  In  part,  at  least,  these  efforts  depend  on  the 
kind  of  influence  the  executive  has  upon  those  whose  work  his  or 
her  own  behavior  touches.  The  executive's  own  behavior 
contributes  to  the  achievement  of  organizational  goals  only  by 
its  influence  on  the  perceptions,  attitudes,  and  motives  of  other 
people  in  the  organization  and  on  their  subsequent  behavior.  In 
addition,  achieved  successes  or  experienced  failures  depend  on 
much  more  than  internal  organizational  phenomena.  Economic 
conditions,  actions  by  opponents/competitors,  availability  of 
funds  or  resources  all  affect  organizational  outcomes.  In  many 
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cases,  these  outcomes  might  be  modified  but  not  necessarily 
reversed  by  executive  actions. 

If  we  would,  nonetheless,  decide  to  evaluate  executives  on 
the  basis  of  externally  observable  outcomes,  we  should  be 
"systematic"  in  our  approach.  Hopefully,  our  methods  could  be 
parsimonious.  Hopefully,  we  might  be  able  to  focus  on  only  one 
or  very  few  criteria.  How  many  outcomes  are  there?  Factor 
analytic  techniques  to  ascertain  "external"  measurement 
reqv  rements  have  been  employed  to  answer  that  question. 

According  to  Campbell,  Dunnette,  Lawler,  and  Weick  (1970), 
factorial  studies  of  organizational  optimization  have  identified 
ten  independent  performance  dimensions.  These  are  (1)  business 
volume,  (2)  production  costs,  (3)  index  of  new  agent 
productivity,  (4)  index  of  relative  usefulness  of  agents,  (5) 
business  mix,  (6)  manpower  growth,  (7)  managers'  commissions,  (8) 
maintenance  costs,  (9)  agent  productivity,  and  (10)  market 
penetration.  These  dimensions  might  be  used  as  criteria  against 
which  executive  performance  appraisal  decisions  can  be  made. 

According  to  Campbell,  et.  al.,  global  measures  of 
managerial  effectiveness  can  also  be  used  as  criteria  for 
assessing  managers.  One  widely  used  procedure  consists  simply  of 
rankings  by  former  superiors  on  total  managerial  effectiveness. 
Others  include  salary  and  organizational  level  indices,  corrected 
statistically  for  age  or  length  of  time  in  an  organization. 

These  measures  constitute  broad  conceptions  of  success,  but  have 
the  added  advantage  of  reflecting  pooled  estimates  of  many 
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superiors’  judgments  over  an  individual  manager's  total  career. 
Such  indices,  thus,  constitute  a  summation  of  an  executive's 
ability  to  optimize  organizational  systems  and  to  perform  crucial 
managerial  job  activities.  Of  course,  observation  of  such  broad 
measures  may  suffer  from  an  inability  to  identify  as  fully  as 
possible  all  behaviors  judged  to  be  relevant  to  the  conceptual 
criterion.  A  variety  of  issues  serve  to  contaminate  this 
approach.  Included  among  these  are  observational  errors  such  as 
response  sets,  chance  response  tendencies  and  differential 
observer  characteristics  or  perspectives,  e.g.,  those  which  occur 
among  peers,  subordinates  and  superiors. 

Clearly,  the  external  criterion  approach  has  not  provided  us 
with  a  parsimonious  method  of  executive  assessment.  Should 
executives  only  be  selected  if  they  are  competent  in  all  these 
areas  of  endeavor?  Should  they  be  trained  in  all  of  them?  Do 
different  executive  tasks  or  different  executive  positions 
require  different  combinations  of  expertise?  We  don't  have  final 
answers  to  these  questions.  But,  possibly  the  most  serious  flaws 
of  external  "results"  measurement  are  the  confounding  effects  of 
specific  executive  task  environments.  It  is  well  known  that 
corporate  vice  presidents  who  are  placed  in  charge  of  a  division 
during  a  recession  are  often  fired,  while  those  who  take  control 
of  that  division  at  the  time  of  an  economic  upswing  may  find 

themselves  promoted  or  may  be  hired  as  President  of  another 
company.  In  other  words,  we  need  more  than  mere  perceptions  or 


measurements  of  outcome.  At  least  in  addition,  we  need  to  know 


what  decisions  were  made  by  an  individual  executive,  why  they 
were  made  and  how  they  were  made.  We  need  to  know  more  about  the 
specific  style  of  the  manager.  We  need  to  know  how  a  manager 
interfaces  that  style  with  changing  task  demands. 

The  interest  in  individual  difference  charac terist ics  that 
may  motivate  and  control  executive  behavior  has  resulted  in 
attempts  to  engage  in  systematic  mul tiple  measurement  of 
personality  and  stylistic  characteristics  of  managers.  Entire 
units  within  organizations  as  well  as  independent  corporations 
whose  purpose  it  is  to  serve  other  organizations  have  grown  up 
around  the  multiple  measurement  notion.  Their  sole  purpose  may 
be  the  prediction  and  training  of  executive  information  handling 
and  performance.  Many  of  these  efforts  are  now  handled  in 
"Assessment  Centers." 

According  to  Finkle  (1976),  the  original  model  for 
assessment  centers  was  generated  by  American  Telephone  and 
Teleg,raph  Company  in  1956.  Subsequently,  many  large 
organizations,  including  most  Fortune  500  corporations,  have 
employed  this  technique.  The  assessment  center  approach  is 
generally  distinguished  from  other  managerial  assessment 
techniques  by  the  use  of  a  fixed  size  group  of  assessees. 
Typically,  the  choice  size  for  this  group  is  approximately  12 
individuals.  Another  unique  characteristic  is  the  use  of 
multiple  assessment  methods.  Typically,  these  include  such 
measures  as:  objective  tests,  projective  tests,  interviews, 
situational  exercises  and  peer  ratings,  among  others.  Typical 


objective  tests  may  be  such  techniques  as:  the  Miller  Analogies 
Test,  the  Watson-Glaser  Critical  Thinking  Test,  the  Guilford 
Martin  Personality  Inventory,  the  Edwards  Personal  Preference 
Scale,  and  the  Allport-Vernon-Lindsey  Scale  of  Values,  among 
others.  Projective  instruments  often  include  the  Thematic 
Apperception  Test  as  a  standard  technique. 

Assessment  centers,  of  course,  both  benefit  and  suffer  from 
the  specific  techniques  that  they  utilize.  Their  approach  of 
measuring  individual  competence,  its  emergence  and  application  in 
task  group  settings  is  excellent  in  conceptualization. 
Unfortunately,  the  techniques  used  by  different  assessment 
centers  differ  widely,  making  comparative  evaluations  on  an 
overall  basis  difficult.  Key  validity  studies  have  demonstrated 
the  usefulness  of  certain  assessment  center  techniques  for 
specific  organizations.  However,  less  is  known  about  the  general 
applicability  of  assessment  center  results  to  a  variety  of 
corporate  and  military  task  requirements.  As  Finkle  (1976)  has 
stated  : 

"The  most  critical  factor  in  further  applications  (of 
assessment  centers)  would  clearly  seem  to  be  the  need  of 
continued  close,  qualified,  professional  guidance  and  ad-hoc 
verification  and  control....  Given  such  professional 
attention,  the  theme  concepts  of  the  assessment  cente: 
package  can  and  will  emerge  as  helpful  and  (contain)  much 
needed  input  to  critical  organizational  manpower  decisions." 

What  Should  be  Measured? 

The  value  of  the  assessment  center  approach  is  its  focus  on 
the  existence  and  application  of  individual  competence.  But,  for 
example,  will  the  Miller  Analogies  Test,  will  participation  in 


group  task  performance  reveal  the  important  characteristics  of  a 
manager,  i.e.,  the  characteristics  that  lead  to  organizational 
success?  Will  obtained  scores  predict  excellence  across  a  wide 
variety  of  tasks?  Will  they  be  sensitive  to  the  capacity  for 
change  when  task  environments  change?  The  potential  exists. 
However,  generally,  that  important  goal  is  not  attained.  That 
goal  has  been  especially  elusive  in  the  assessment  and  training 
of  senior  executives  and  of  those  who  would,  some  day,  advance  to 
senior  levels. 

The  problem,  again,  is  found  in  the  absence  of  appropriate 
and  meaningful  criteria.  To  determine  what  makes  excellent 
senior  executives  differ  from  mediocre  senior  executives,  we  must 
first  identify  the  underlying  causes  of  those  differences.  We 
must  look  beyond  ratings,  intellectual  capacity,  interpersonal 
skill  and  many  of  the  other  values  that  are  obtained  via  the 
usual  assessment  methods.  We  must  go  beyond  descriptive  terms 
such  as  "intuition"  (Isenberg,  1984)  that  claim  to  "explain" 
these  differences.  We  should  clarify  and  achieve  a  basic 
understanding  of  certain  characteristics  that  are  ascribed  to 
excellent  senior  executives  such  as  "strategic  opportunism" 
(Isenberg,  1986).  Indeed,  eminently  successful  and  not  so 
successful  executives  differ  in  their  application  of  strategic 
opportunism.  But,  what  is  it?  What  kind  of  capacity,  what  kind 
of  thinking,  what  kind  of  information  processing  underlies  that 
form  of  "excellence,"  especially  at  senior  levels? 


The  majority  of  "individual"  assessment  techniques  and  their 
associated  training  methodologies  tend  to  either  focus  on  a  very 
"basic"  level  of  analysis  or  employ  rather  complex  procedures 
such  as  "free"  simulations  (cf.  the  definitions  and  distinctions 
provided  by  Fromkin  and  Streufert,  1976,  1983;  Streufert  and 
Swezey,  1986).  The  more  basic  assessment  methods,  including 
objective  and  projective  tests  (as  well  as  most  interviews),  tend 
to  provide  ample  information  about  motivation,  technical  skill 
level,  experience,  native  intelligence  and  about  trained 
competence  to  deal  with  specific  events.  To  the  degree  to  which 
obtained  information  can  be  considered  an  accurate  reflection  of 
an  executive's  capacity,  that  information  is  certainly  useful, 
especially  at  junior  levels.  But  it  is  not  enough.  At  higher 
executive  levels,  the  problem  is  compounded.  As  already  stated 
above,  senior  level  executives  both  in  private  industry  and  in 
the  military  do  not  differ  greatly  in  such  characteristics  as 
motivation,  intelligence  and  experience.  Something  else  affects 
their  level  of  competence. 

In  contrast,  then,  measuring  and  training  performance  in 
highly  complex  task  settings,  e.g.,  via  free  simulations,  appears 
to  be  more  appropriate.  But  is  it?  Not  always.  Certainly 
"real"  performance  in  a  potentially  quite  realistic  miniature 
task  of  sufficient  complexity  is  observed,  assessed  and 
potentially  trained.  However,  the  typically  employed  free 
simulation  techniques  still  suffer  from  criteria  problems.  Since 
each  executive  or  team  of  executives  make(s)  decisions  that 


affect  their  environment  "down  the  line,"  each  is,  in  fact, 
exposed  to  a  different  developing  task  environment.  The 
performance  of  different  individuals  and  different  groups  cannot 
be  meaningfully  compared.  Systematic  comparative  analysis  is  not 
possible.  Task  demands,  modified  by  the  actions  of  participating 
executives  themselves,  may  be  more  or  less  demanding  and  may 
involve  diverse  levels  of  uncertainty  and  complexity.  In  other 
words,  the  obtained  data  would  only  provide  limited  insights  into 
the  potential  performance  capacity  of  evaluated  executives.  That 
would  especially  be  the  case  where  assessment  is  to  be  valid  or 
training  is  to  be  applicable  to  a  wide  variety  of  fluid  task 
requirements  and  situational  conditions. 

Any  more  useful,  yet  adequately  complex  technique  to  assess 
executive  competence  and  to  predict  executive  performance  would 
have  to  operate  at  a  "middle  ground"  where  executives  would 
function  in  a  controlled  (more  or  less  constant  across 
individuals  or  groups)  task  environment  that  would,  nonetheless, 
closely  approach  the  multifaceted,  uncertain  and  fluid  tasks  that 
senior  executives  may  face.  The  technique  would  have  to  permit 
measurement  of  "what"  executives  do  in  response  to  events  and 
task  demands.  It  would  also  have  to  measure  "how"  executives 
approach  their  jobs.  It  would  have  to  assess  how  executives 
think,  how  they  obtain  and  deal  with  information  and  how  they  go 
about  making  decisions,  either  as  individuals  or  as  participants 
in  task  groups.  The  technique  would  have  to  measure  how 
strategies  are  developed,  applied  and  tied  to  goals.  It  would 


have  to  consider  goal  structure.  It  would  have  to  measure 
flexibility  of  thought,  of  information  orientation,  of  decision 
making,  of  strategy  development/application ,  and  of  goal 
orientation.  Most  of  all,  the  method  must  be  able  to  measure 
these  varying  aspects  of  executive  functioning  reliably.  It  must 
achieve  a  level  of  validity  that  permits  meaningful  predictive 
assessment  and  training.  Finally,  the  technique  must  extend 
beyond  the  contributions  to  executive  performance  that  are  due  to 
"experience",  "intelligence",  "motivation",  and  standard  forms  of 
"training"  (including  educational  level). 

Streufert  and  associates  (Streufert,  in  press;  Streufert  and 
Swezey,  1986)  have  discussed  a  q uas i -ex per i men t a  1  simulation 
technique  that  fulfills  these  requirements.  This  assessment  and 
training  technique  was  based  on  the  early  development  of 
experimental  simulation  techniques  by  Streufert,  Clardy,  Driver, 
Karlins,  Schroder  and  Suedfeld,  1965  and  by  Streufert,  Kliger, 
Castore  and  Driver,  1967.  Experimental  simulations  have  been 
utilized  to  obtain  a  large  quantity  of  data  that  are  concerned 
with  a  number  of  environmental  and  task  effects  (as  well  as 
individual  difference  effects)  upon  a  variety  of  performance/ 
decision  making  variables  (cf.  e.g.,  Streufert,  1970;  Streufert 
and  Streufert,  1978).  The  multiple  information  gained  in  these 
research  projects  was  useful  in  efforts  to  tailo.  a  quasi- 
experimental  simulation  methodology  toward  more  optimal  executive 
assessment  and  training.  In  its  most  recent  version,  the  quasi- 
experimental  simulation  technique  provides  for  the  (automated) 


measurement  of  some  sixty  aspects  of  managerial  decision  making 
performance  (cf.  e.g.,  Streufert,  Pogash  and  Piasecki,  1986) 
which  load  on  ten  orthogonal  performance  factors  (see  below). 

Two  simulation  scenarios  are  presently  available.  Inter¬ 
simulation  measurement  reliability  for  structural  measures  (how 
executives  deal  with  their  tasks)  varies  from  +.65  to  +.95 
(Streufert,  Pogash,  Piasecki,  Hunter  and  Repman,  1986).  We  will 
initially  consider  the  factor  structure  of  our  simulation  based 


measurement . 


FACTOR  STRUCTURE 


In  previous  reports,  we  have  provided  extensive  information 
about  our  simulation  based  measurement  techniques  (formulas)  and 
about  the  reliability  of  those  measures  (Streufert,  Pogash  and 
Piasecki,  1986;  Streufert,  Pogash,  Piasecki,  Hunter  and  Repman, 
1986).  Since  the  completion  of  these  earlier  reports,  four 
additional  measures  of  decision  making  performance  have  been 
added  to  the  simulation  software.  To  begin  with,  let  us  consider 
the  characteristics  of  those  four  measures. 

Additional  Measurement 

Among  the  measures  discussed  in  earlier  reports,  several 
were  designed  to  assess  strategic  capacity.  The  most  basic  of 
those  measures,  "Number  of  Integrations,"  considered  the 
frequency  of  single  strategic  interconnections  between  any  two 
sequential  actions.  I_f  an  initial  action  was  taken  to  make  a 
subsequent  action  possible,  and  i_f  that  later  action  was  actually 
carried  out,  a  single  "simple"  strategic  intent  war  scored. 
Obviously,  the  simulation  pa r t ic  i  pa n t ( s  )  had  engaged  in  one  step 
of  forward  planning.  In  contrast,  if  simulation  pa r t ic i pan t ( s  ) 
made  use  of  a  previous  action  in  a  later  action  without  any 
earlier  intent  to  engage  in  that  subsequent  action,  credit  for 
opportunistic  (backward)  thinking  was  given.  Measure  4  of  our 
simulation  technology  assesses  strategic  "Forward  Integrations." 
Measure  7  assesses  opportunistic  "Backward  Integrations." 

The  single  step  strategic  measurement  reflected  in  Measure  4 
(Forward  Integration)  has  provided  the  foundation  of  more  complex 


their  formula.  In  other  words,  each  of  the  measures  of  more 
complex  strategic  performance  is  intentionally  confounded  with 
the  Number  of  Integrations  measure.  Further,  each  higher  level 
measure  is  intentionally  confounded  with  the  next  lower 
antecedent  measure.  For  example,  QIS  contains  the  formula  for 
Time  Weight,  WQ IS  contains  the  formula  for  QIS  and  so  forth.  For 
most  purposes  such  confounding  is  useful.  Obviously,  any  higher 
level  of  strategy  cannot  be  accomplished  without  the  capacity  for 
the  next  lower  level.  For  example,  measuring  time  length  of 
strategic  (integrated)  planning  is  meaningless  if  no  strategic 
plans  were  made  in  the  first  place. 

However,  as  our  strategy  measurement  was  previously 
designed,  the  measures  told  us  little  about  the  added  competence 
represented  by  each  higher  level  strategy  measurement  procedure. 
An  understanding  of  added  competence  can,  for  example,  be  highly 
useful  in  the  development  and  evaluation  of  theory,  especially 
theories  that  are  based  on  observations.*  Consider  the  view  of 
Streufert  and  Swezey  (1986)  who  have  argued  that  the  number  of 
steps  of  strategic  planning  capacity  are  predictive  of  executive 
competence.  In  contrast,  Jaques  (1976)  has  suggested  that  the 
maximum  time  length  of  planning  predicts  managerial  competence. 
Our  previous  measures  were  not  able  to  distinguish  between  the 
number  of  steps  and  the  time  involved  in  utilizing  those  steps. 

To  determine  whether  either  Jaques  or  whether  Streufert  and 

Observations  necessarily  confounded  higher  order  strategies 
with  their  lower  order  antecedents. 


Swezey  are  correct,  a  clear  measurement  distinction  between 
stepwise  and  time  based  planning  must  be  made.  In  other  words, 
we  must  consider,  for  example,  the  specific  component 
contributions  that  are  inherent  in  the  Time  Weight  Measure 
(representing  at  least  part  of  the  views  advanced  by  Jaques)  as 
well  as  those  that  are  inherent  in  Streufert  and  Streufert's 
(1978)  QIS  measure  (representing  at  least  part  of  the  views  of 
Streufert  and  Swezey).  In  the  view  of  the  latter  authors, 
extensive  time  length  of  planning  may  be  counterproductive  for 
executive  performance  (cf.  also  the  work  of  Isenberg,  e.g., 

1984).  Indeed,  Jaques'  capacity  to  predict  managerial  competence 
on  the  basis  of  the  time  length  of  planning  might  be  due  to  the 
confounding  of  time  length  with  planning  frequency  (e.g.,  the 
number  of  steps  needed  to  achieve  a  long-range  goal). 

To  separate  the  predictive  component  s  of  various  measures  of 
strategic  competence  (strategic  action  in  the  simulations),  i.e., 
to  calculate  the  addi tional  capacity  for  strategic  performance 
inherent  in  each  higher  order  measure,  four  new  formulas  were 
developed.  Each  of  these  formulas  divides  the  obtained  value  of 
our  previous  measures  of  strategic  performance  (e.g.  QIS)  by  the 
next  lower  level  of  strategic  assessment  (e.g.,  Time  Weight). 

(In  addition,  each  of  these  formulas  can  be  expressed  as  a 
proportion  of  general  activity,  i.e.,  the  number  of  decisions 
made.)  The  following  four  measures  were  added  as  part  of  the 
revised  simulation  software: 


Measure  5Q:  Component  Contribution  by  Multiplexity  F 
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Measure  6Q:  Component  Contribution  by  Integration  Time 
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Measure  9Q:  Component  Contribution  by  QIS 
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*The  formulas  might  have  been  mathematically  simplified  for 
this  report.  However,  they  are  provided  in  a  form  that  clarifies 
their  derivation.  For  an  explanation  of  terms,  the  uninitiated 
reader  is  referred  to  Streufert,  Pogash  and  Piasecki  (1986). 


Measure  10Q:  Component  Contribution  by  WQIS 
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Factor  Extraction 

Factor  analysis  of  performance  scores  obtained  in  111 
simulations  was  based  on  the  i n t e r c or r e 1  a t  i  ons  of  48  measures. 

The  measures  are  (1)  those  discussed  in  the  report  by  Streufert, 
Pogash  and  Piasecki,  (1986),  (2)  the  four  Q  measures  discussed 
above,  and  (3)  11  quotients  describing  the  proportional  incidence 
(P  scores)  of  performance  frequencies.  P  scores  were  calculated 
by  dividing  a  relevant  measure  score  by  the  number  of  decisions 
made  (actions  taken)  during  the  time  period  addressed  by  that 
particular  measure  based  score.  The  obtained  "P  scores"  are 
identified  by  the  basic  measure  number  (see  Streufert,  Pogash  and 
Piasecki,  1986)  to  which  they  apply:  that  is  as  Measures  4P,  7P, 
8P  ,  1 7 P  ,  19P,  20P ,  2 1 P ,  2 2 P ,  27P,  28P  and  36P. 

Twelve  factors  accounted  for  85.4%  of  the  variance  in  the 
data.  Lower  order  factors  tended  to  reach  asymptote  in  their 
Eigenvalues  and  were  ignored.  The  variance  accounted  for  by  each 
of  the  largest  12  factors  is  presented  in  Table  1.  Factor 
Interpretation  was  based  on  Varimax  rotation  procedure.  Analysis 
utilized  a  SPSS-PC  +  programs.  Each  factor  is  discussed  below  in 


TABLE  1 


Factor 


PERCENT  OF  VARIANCE  ACCOUNTED 
FOR  BY  TWELVE  FACTORS 


Percent  of 

Eigenvalue  Variance 


Cumulat  i  ve 
Percent 


order  of  the  amount  of  variance  accounted  for: 


Factor  1:  Simple  Strategy 

P 

The  largest  single  factor  is  defined  by  highest  loadings  on  J 

i 

variables  4P  (Number  of  Integrations  in  Proportion  to  Number  of  ! 

i 

Decisions,  loading  +.90)  and  22P  (Integrations  within  Decision 
Categories  in  Proportion  to  Number  of  Decisions,  loading  .89). 

Other  measures  loading  highly  on  this  factor  include  36P,  5Q,  4, 

20P ,  22,  36  and  19P  (all  above  .8),  21P,  21,  20,  34  and  19  (above  • 

.7),  32,  35  and  7P  (above  .6),  7,  13,  12  and  2  (above  .4).  ; 

General  Unintegrated  Decision  Making  loaded  negatively  (Measure  ! 

14,  loading  =  -.49). 

While  some  measures  of  higher  order  strategy  do  load  on  this  ' 

factor,  including  one  of  the  Q  measures,  the  defining  measures 

t 

(as  well  as  the  majority  of  other  measures  which  relate  to  this 
factor)  reflect  simple ,  more  likely  single  step  or  short-term 
strategic  actions.  j 

Factor  2:  Activity 

The  second  factor  is  defined  by  scores  for  Measures  1 

f 

(Number  of  Decisions  Made,  loading  .95)  and  3  (Number  of  , 

Decision/ Action  Categories,  loading  .91).  Measure  17,  (Number  of  ) 

Decisions  in  Response)  generated  a  loading  of  .77,  with  Measures  \ 

20  (Total  Forward  Integrative  Activity),  19  (Total  Integrative  | 

I 

Activity),  7  (Number  of  Backward  Integrations),  12  (Planned 
Integrations),  8  (Number  of  Unintegrated  Decisions),  and  2  ; 

(Number  of  Respondent  Decisions)  loading  between  .4  and  .6.  The  ; 

activity  focus  of  this  factor  is  undeniable.  All  variables  that 


involve  general  frequency  counts  of  activity  are  represented  at 
levels  of  at  least  .4. 


Factor  3:  Low  Level  Performance 

Factor  3  is  defined  by  a  loading  of  .89  on  Measure  8P  and 
.79  on  Measure  8,  reflecting  Unintegrated  (i.e.,  non-st rategi c ) 
Decision  Making.  Also  represented  are  Measures  2,  Respondent 
Decisions,  and  7  Backward  (i.e.,  unplanned  opportunistic) 
Integrations  at  loads  above  .6.  Measures  13,  Planned  (but 
unfinished)  Strategic  Integrations,  and  •  ,  Total  Integrative 

Activity  both  load  above  .4.  The  inclusion  of  Measure  19  on  this 
factor  was  generated  by  the  contributions  of  Measures  7  and  13  to 
this  score.  It  should  be  noted  that  the  TFIA  Measure  20  did  not 
load  meaningfully  on  Factor  3,  despite  the  fact  that  the  latter 
measure  includes  the  contribution  of  planned  but  incomplete 
actions  (cf.  Measure  13).  In  other  words,  we  are  dealing  with 
rather  simple  but  possibly  quite  "decisive"  responses  to  the 
simulated  task. 

Factor  4:  Complex  Strategy 

The  defining  measure  on  this  factor  is  Measure  23, 

Proportion  of  Category  Integrations  (loading  .80).  This  measure 
reflects  the  level  of  preponderance  of  strategic  actions  that 
cross  widely  disparate  action  areas  as  compared  to  strategic 
actions  that  address  related  and  similar  areas  of  activity. 
Persons  scoring  high  on  Measure  23  tend  to  have  a  broad  overview 
and  use  their  understanding  to  generate  a  more  encompassing 
strategic  approach  to  executive  actions. 


Other  measures  loading  on  this  factor  include  (not 


surprisingly)  21  (.45)  and  21P  (.53)  reflecting  strategic 
integrations  across  diverse  decision  categories.  Measures  28P 
(.61)  and  28  (.56)  reflecting  the  integrative  strategic 
application  of  information  obtained  through  previous  search  as 
well  as  two  of  the  Q  measures.  Measure  9Q,  component 
Contribution  by  QIS  loaded  .48.  A  possible  surprise  to  some 
readers  may  be  the  negative  (-.50)  loading  of  Measure  6Q, 
Component  Contribution  by  Integration  Time  Weight.  The  views  of 
Jaques  (1976)  would  likely  have  associated  that  measure 
positively  with  complex  strategic  activity.  Whether  or  not 
Measure  6Q  or  a  factor  loading  heavily  on  6Q  would  predict 
executive  success  will  be  considered  later  in  this  paper. 

Clearly  Factor  4  describes  individuals  who  possess  a  broad 
overview  and  who  interrelate  quite  diverse  task  components  to 
generate  an  encompassing  strategy. 

Factor  5;  Hierarchical,  Analytic  and  Inflexible  Strategy 

This  factor  is  defined  by  negative  loadings  on  Measures  24 
(-.91)  and  25  (-.89).  Both  of  these  measures  assess  the  capacity 
to  shift  between  more  decisive  and  more  planned  strategic  actions 
as  task  demands  change.  The  persons  defined  by  the  obtained 
negative  loadings  on  this  factor  are  not  able  to  adapt  their 
style  to  variations  in  task  demands.  They  may,  on  the  other 
hand,  possibly  be  persons  who  can  plan  long  into  the  future 
(Measure  6Q  loads  at  -.36).  However,  these  individuals  seem  to 
differ  greatly  from  individuals  who  develop  sequential  stepwise 


and  complex  strategies  (Measures  9Q,  loading  .35  and  10Q,  loading 
.59  suggest  the  absence  of  Quality  of  Integrated  Strategies  at 
all  levels). 

Factor  6:  Sensitivity  to  Emergencies 

Factor  6  generated  the  highest  loading  for  Measure  33, 

Number  of  Decisions  Made  in  Response  to  an  Emergency,  loading 
.80.  Measures  31,  Integrated  Responses  to  the  Emergency,  and  32, 
Applied  Actions  Related  to  the  Disaster,  loaded  between  .65  and 
.70.  Measure  35,  Total  Integrative  Activity  Related  to  the 
Emergency  produced  a  value  of  .47.  A  negative  loading  of  -.43  on 
Measure  29,  Disaster  Response  Speed,  indicates  that  responses  to 
the  advent  of  the  emergency  were  generated  quickly  and  likely 
decisively. 

Factor  7:  Information  Search  Activity 

Factor  7  is  defined  by  loadings  of  .84  on  Measure  27P  and 
.83  on  Measure  27,  reflecting  the  frequency  with  which  simulation 
participants  engaged  in  Information  Search.  The  utilization  of 
previous  search  in  strategic  activity  at  a  later  time  is 
reflected  in  loadings  of  .46  on  Measure  28P  and  .44  on  Measure 


Factor  8:  Spread  Across  Decision  Categories 

This  factor  was  generated  by  extremely  high  loadings  (.96 
each)  on  two  highly  similar  measures  (15  and  16)  and  represents 
only  a  reflection  of  that  measure.  The  factor  will  be  ignored 
because  of  its  identify  with  a  measure,  i.e.,  its  limited 


implications  for  measure  commonality. 


Factor  9 


Factor  9  is  defined  by  Measures  18,  Most  Recent  Response 
Speed,  loading  .91,  and  Measure  11,  Average  Response  Speed, 
loading  .85.  While  the  first  of  these  measures  assesses 
the  time  between  the  receipt  of  information  and  the  first  action 
responding  to  that  information,  the  second  considers  all  relevant 
responses  and  their  average  distance  in  time  to  the  receipt  of 
information.  Persons  with  high  scores  on  this  factor  would 
likely  delay  action;  persons  with  low  scores  would  likely  respond 
rapidly  to  information  receipt. 

Factor  10:  Emergency  Response  Speed 

Loadings  of  .76  on  Measure  30,  Average  Disaster  Response 
Speed,  and  .56  on  Measure  29,  First  Respondent  Decision  in 
Response  to  information  communicating  an  Emergency  suggest  that 
we  are  dealing  with  the  rapidity  of  handling  a  serious  emerging 
problem.  A  loading  on  this  factor  for  Measure  1 7P  (-.65)  is  also 
of  interest.  It  is  possible  that  persons  who  respond  immediately 
(decisively?)  to  a  disaster  make  more  respondent  decisions 
overall. 

Factor  11;  Recovery  1 

Factor  11  is  primarily  defined  by  Measure  26,  Recovery  1, 
loading  .84.  That  measure  reflects  the  degree  to  which 
simulation  participants  were  able  to  re-establish  strategic 
planning  after  an  emergency  had  been  dealt  with.  Lesser  loadings 
on  Measures  28  (.33)  and  28P  (.25)  probably  imply  that  the 


strategic  (integrated)  utilization  of  previously  obtained 


information  may  be  an  important  ingredient  in  recovery  from  more 
applied  and  immediate  responses  to  a  temporary  emergency. 

Factor  12;  Recovery  2 

With  only  a  single  (negative)  loading  (-.85)  on  a  single 
measure  (#37,  Recovery  2),  this  factor  had  little  value  beyond 
the  score  obtained  for  the  Measure  it  describes.  Consequently 
the  factor  will  be  ignored. 
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VALIDITY 

Indicators  of  Success:  A  Correlational  Analysis 

The  scores  obtained  in  111  completed  simulations  reflect  the 
performance  of  a  variety  of  diverse  individuals.  While  all 
simulation  participants  had  at  least  some  decision  making  or 
managerial  experience,  the  level  and  characteristics  of  that 
experience  was  quite  disparate.  To  obtain  indications  of 
validity  across  such  a  diverse  sample  of  participants,  relatively 
non-specific  indicators  of  attained  success  were  chosen  as 
potential  bases  for  a  validity  analysis.  Among  them  are  measures 
of  job  and  income  characteristics.  In  addition,  other 
potentially  interesting  variables  describing  respondent 
characteristics,  e.g.,  satisfaction  measures,  were  added.  The 
following  descriptors  of  simulation  respondents  were  assessed 
outside  of  the  simulation  environment: 

1  .  Job  Success 

2.  Occupational  Level 

3.  Number  of  Persons  Supervised 

4.  Length  in  Present  Position  (negative) 

5.  Number  of  Job  Changes  in  the  Last  10  Years 

6.  Job  Satisfaction  (on  a  scale  from  1-low  to  7-high) 

7.  Job  Level  Satisfaction  (1-7  scale) 

8.  Satisfaction  with  Pay  (1-7  scale) 

9.  Personal  Income 

10.  Number  of  Changes  of  Organizations  in  10  Years 


11.  Number  of  Promotions  in  10  Years 


12.  Personal  and  Spouse  Income  Combined 

13.  Educational  Level 

Some  of  the  13  measures,  obtained  from  responses  to  a  biograph¬ 
ical  inventory,  were  expected  to  be  partial  but  direct  indicators 
of  success.  Others  (e.g.,  Educational  Level)  were  expected  to  be 
independent  contributors  to  success  that  might  have  their  own 
impact  on  managerial  competence.  Adjustment  of  scores  for  age 
and  job  type  were  based  on  values  (e.g.,  income  levels)  obtained 
from  the  Bureau  of  Labor  Statistics  (U.S.  Government). 

Correlations  between  biographic  indicators  and  a  number  of 
simulation  measures  were  calculated.  Simulation  measures  were 
selected  for  this  purpose  if  they  loaded  highest  or  highly  on  any 
one  of  the  previously  discussed  factors  or  if  they  were  of 
interest  for  theoretical  reasons.  For  example,  Measure  6Q  (see 
the  discussion  of  added  measures  below)  was  included  for 
theoretical  reasons  because  of  its  importance  to  the  views  of 
Jaques  versus  Streufert  and  Swezey  .  The  following  measures  were 
intercorrelated  with  biographical  indicators: 

1.  Measure  4  P :  Number  of  Integrations  as  a  Proportion  of 
Decisions  (measuring  simple  strategy). 

2.  Measure  22P:  Integrations  within  Categories  as  a 
Proportion  of  Decisions  (measuring  strategy  within  a 
specific  focus  area), 

3.  Measure  1:  Number  of  Decisions. 

4.  Measure  3:  Number  of  Decision  Categories  (a  measure  of 
the  different  kinds  of  actions  taken). 

Measure  8P:  Number  of  Unintegrated  Respondent 
Decisions  as  a  Proportion  of  the  Number  of  Decisions  (a 
measure  of  the  extent  of  rather  simplistic 
functioning  )  . 


5. 


Measure  6Q:  Specific  Contributions  by  the  Time  Length 
of  Planning  (a  measure  reflecting  the  characteristic 
which  reflects  Jaques  views  of  executive  success). 

Measure  23:  Proportion  of  Category  Integrations  (a 
measure  of  the  breadth  of  a  strategic  approach). 

Measure  24:  Emergency  Shift  1  (an  assessment  of 
decision  maker  capacity  to  shift  from  an  integrated 
strategic  mode  of  action  to  a  respondent  mode  whenever 
task  demands  require). 

Measure  25:  Emergency  Shift  2  (Measure  24  corrected 
for  the  actual  amount  of  effort  expended  on  decision 
making  activity  which  is  related  to  an  emergency). 

Note  that  both  measures  24  and  25  produce  1 ow  scores 
where  performance  is  excellent,  i.e.,  negative 
correlations  would  be  expected  if  these  measures  are 
predictive  of  success. 

Measure  33:  Number  of  Disaster  Decisions  (a  measure  of 
activity  that  is  directly  related  to  handling  an 
emergency )  . 

Measure  27P:  Number  of  Information  Search  Decisions  as 
a  Proportion  of  the  Number  of  Decisions. 

Measure  11:  Average  Response  Speed  (a  measure  of  the 
mean  time  distance  between  relevant  actions  and 
previously  received  information. 

Measure  18:  Most  Recent  Response  Speed  (the  rapidity 
of  response  to  incoming  information. 

Measure  29:  Disaster  Response  Speed  (the  rapidity  of 
response  to  information  that  is  communicating  an 
emergency  ) . 

Measure  30:  Average  Disaster  Response  Speed  (a  measure 
of  the  mean  time  distance  between  relevant  actions  and 
the  receipt  of  information  about  an  emergency). 

Measure  26:  Recovery  1  (a  measure  of  the  capacity  to 
regain  a  strategic  orientation  after  problems 
associated  with  an  emergency  have  been  resolved). 

Measure  5:  Multiplexity  F:  (a  measure  of  stepwise 
forward  strategic  planning,  without  consideration  of 
the  time  length  of  planning). 


18.  Measure  5Q:  The  specific  contributions  of  the 
complexity  of  stepwise  forward  planning  without 
consideration  of  its  frequency. 

19.  Measure  9:  Quality  of  Integrated  Strategies  (a  measure 
of  the  overall  complexity  of  forward  planning  in  the 
past,  the  present,  and  toward  the  future  as  well  as  the 
degree  of  interrelatedness  of  multiple  strategic 

plans )  . 

20.  Measure  9Q:  The  specific  contributions  of  the 
complexity  of  previous  and  current  forward  planning 
obtained  via  Measure  9  without  consideration  of  the 
frequency  and  time  length  of  strategic  planning. 

21.  Measure  10Q:  The  specific  contribution  of  the 
complexity  of  sequential  chains  of  planning  as  obtained 
via  Measure  9Q  beyond  any  three  sequential  steps  in  a 
strategic  sequence. 

The  intercorrelations  between  biographical  indicators  and 
simulation  measures  as  well  as  any  obtained  significance  levels 
for  those  intercorrelations  are  obtained  in  Table  2.  \  view  of 

the  results  suggests  that  the  most  significant  i n t er c or r e 1  a t i on s 
with  simulation  measures  are  obtained  for  Income  Level,  with 
additional  useful  data  obtained  for  Occupational  Level  and,  to 
some  extent,  for  the  Number  of  Persons  Supervised.  To  present  a 
more  easily  read  listing  of  results  Table  3  was  prepared.  That 
table  lists  the  significance  levels  represented  by  some  of  the 
intercorrelation  between  simulation  measures  and  the  three  most 
important  indicators  of  (biographically  obtained)  success. 

One  additional  interesting  finding  that  suggests  a  focus  for 
future  research  should  be  noted.  Significant  positive 
correlations  between  such  biographic  success  variables  as 
Personal  Income  and  measures  of  strategic  performance  such  as 
Number  of  Integrations  were  often  associated  with  significant 


INTERCORRELATIONS  OF  BIOGRAPHIC  AND  SIMULATION  PERFORMANCE  MEASURES 
CORRELATION  COEFFICIENTS  TO  2  DIGITS  WITHOUT  THE  DECIMAL  POINT 


Negative  Intercorrelatlons  were  obtained  and  expected  for  Measures  6Q,  8Pt  24  and  25.  r  •  13  <  .  10+ 

r  -  16  <  .05* 


TABLE  3 


SIGNIFICANCE  LEVELS  OBTAINED  FOR  THE  INTERCORRELATIONS 
OF  SIMULATION  MEASURES  AND  BIOGRAPHIC  INDICATORS 

OF  SUCCESS 


1.  CORRELATIONS  WITH  PERSONAL  INCOME 


Significance 

Level 


Simulation  Measure  _ p  < 

4P  Number  of  Integrations  .05 

8P  Unintegrated  Respondent  Decisions 

(negative)*  .05 

6Q  Time  Length  of  Planning  (negative)*  .10 

27P  Number  of  Information  Search  Decisions  .05 

11  Average  Response  Speed  .05 

18  Most  Recent  Response  Speed  .01 

9  Quality  of  Integrated  Strategies  (QIS)  .10 

9Q  QIS  Contributions  Independent  of  Time 

Length  of  Planning  .05 

29  Disaster  Response  Speed  (corrected  for 

age/job  level)  .01 


2.  CORRELATIONS  WITH  OCCUPATIONAL  LEVEL 

Simulation  Measure  p _ < 

4P  Number  of  Integrations  .01 

33  Number  of  Emergency  Decisions  .01 

9Q  QIS  Contributions  Independent  of  Time 

Length  of  Planning  .05 


Negative  indicates  that  a  negative  correlation  was  expected 
and  obtained. 


3. 

CORRELATIONS  WITH  NUMBER  OF  PERSONS 

SUPERVISED 

Simulation  Measure 

1  1 

Average  Response  Speed 

.05 

18 

Most  Recent  Response  Speed 

.10 

29 

Disaster  Response  Speed 

.05 

9 

Quality  of  Integrated  Strategies 

(QIS) 

.10 

9Q 

QIS  Contributions  Independent  of 
Length  of  Planning 

Time 

.05 

V~»V- 
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nesative  correlations  between  simulation  performance  variables 
and  satisfaction  (for  example  Measure  4P,  Number  of  [strategic 
forward]  Integrations  and  Satisfaction  with  Pay  correlate 
r  =  -.34,  p  <  .01).  On  the  other  hand,  Measure  4P  correlates 
significantly  positive  with  Job  Level,  Income  and  Educational 
Level.  In  part,  these  results  may  be  due  to  the  depressed 
economic  condition  of  Pennsylvania  from  which  the  research 
participants  were  primarily  drawn.  Alternatively,  the  results 
might  reflect  some  specific  characteristic  of  persons  who  do  or 
do  not  possess  the  capacity  to  perform  complex  tasks  in  a  multi¬ 
dimensional  fashion  (cf.  Streufert,  in  press,  Streufert  and 
Swezey,  1986).  Only  subsequent  research  efforts  will  be  able  to 
reach  conclusions  about  the  underlying  basis  of  this  finding. 

The  obtained  data  suggest  that  a  number  of  simulation 
measures  are  meaningfully  (significantly)  related  to  biographic 
indicators  of  success.  It  should  be  noted  that  correlations 
between  simulation  performance  variables  and  educational  level 
also  reached  significance  but  were  not  excessive.  Some 
covariation  between  simulation  performance  and  educational  level 
was  expected  since  the  potential  knowledge  of  the  kinds  of 
actions  that  might  be  combined  in  a  strategy  is,  in  part,  a 
function  of  intelligence,  education  and  experience.  As  Streufert 
(in  press)  has  suggested,  education,  intelligence  and  experience 
provide  basic  building  blocks  of  any  capacity  for  managerial 
success.  However,  those  particular  antecedents  of  adequate 
performance  remain  independent  of  a  multidimensional  capacity  as 


reflected  in  several  simulation  measures.  The  independence  of 
these  predictors  is  especially  evident  at  more  senior  managerial 
levels.  Since  the  subjects  who  participated  in  this  research 
came  from  a  variety  of  employment  levels,  positive  (but  not 
excessive)  correlations  between  measures  of  strateg  c  simulation 
performance  and  measures  of  biographic  success  were  anticipated. 
The  obtained  correlations  between  measures  of  strategy  and 
educational  level  were  also  anticipated.  The  reader  should  note 
that  correlations  for  scores  of  high  level  strategy  (e.g.. 
Measures  9Q  and  10Q)  are  not  significantly  correlated  with 
educational  level  while  scores  of  simple  strategic  responding 
(e.g.,  Measures  3  and  4P)  are. 
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MULTIPLE  REGRESSION 

Through  the  application  of  stepwise  multiple  regression 
procedures,  we  can  determine  the  degree  of  the  contribution  of 
specific  simulation  based  measurement  to  biographic  success.  For 
this  purpose,  separate  regression  equations  were  calculated  for 
applicable  simulation  measures  as  they  predict  Job  Level  and 
Personal  Income.  In  addition,  stepwise  regressions  were 
performed  to  determine  Factor  Score*  contributions  to  Job  Level 
and  Personal  Income.  Only  significant  (beyond  p  =  .05) 
regressions  will  be  reported  in  this  paper. 

Primary  Analysis 

1.  Regression  of  Simulation  Measures  on  Job  Level 

Both  Measure  33  (Number  of  Disaster  Decisions,  an  indicator 
of  relevant  activity  at  the  point  of  an  emergency)  and  Measure  8 
(Number  of  Unintegrated  Respondent  Decisions,  a  measure  of  rather 
simplistic  performance)  predicted  Job  Level.  As  one  might 
expect,  predictions  by  Measure  33  were  positive;  predictions  by 
Measure  8  were  negative  in  nature.  The  obtained  beta  weights  and 
significance  levels  are  presented  in  Table  4. 

2.  Regression  of  Simulation  Measures  on  Personal  Income 

Both  Measure  18  (Most  Recent  Response  Speed,  a  measure  of 
the  rapidity  with  which  a  person  responds  to  incoming  infor¬ 
mation)  and  Measure  28P  (the  Number  of  Integrated  Information 
Search  Decisions  as  a  Proportion  of  the  Total  Number  of 


Factor  scores  for  all  111  participants  were  calculated  for 
the  10  factors  deemed  useful  (see  above). 


TABLE  4 


STEPWISE  REGRESSION  BASED  UPON 
SIMULATION  RAW  DATA 


Prediction 

o  f 

Measure 

Beta  Weight 

Job  Level 

33 

Number  of  Disaster 
Decisions 

.29 

8 

Number  of  Uninte¬ 
grated  Respondent 
Decisions 

-.20 

Personnel 

Income 

18 

Most  Recent 

Response  Speed 

.30 

28P 

Information  Search 
Decisions  Integrated 

.23 

Prediction 

of 

Factor 

Beta  Weight 

Job  Level 

4 

Complex  Strategy 

.19 

Personal 

Income 

9 

Response  Speed 

.30 

7  Information  Search 


51 


40 


r.r.  r. 


Decisions)  predicted  Personal  Income.  Both  predictions  were 
positive  in  nature.  The  beta  weights  and  significance  levels  are 
presented  in  Table  4. 

3.  Regression  of  Factor  Scores  on  Job  Level 

Only  Factor  4  (Complex  Strategy)  was  predictive  of  attained 
job  level.  Apparently  the  simpler  forms  of  strategic  action  and 
the  speed  of  action  represented  in  other  Factors  did  not 
influence  prior  advancement  among  research  participants.  Beta 
weights  and  p  values  are  presented  in  Table  4. 

4.  Regression  of  Factor  Scores  on  Personal  Income 

Personal  Income  Levels  were  predicted  by  Factor  9  (Response 
Speed)  and  Factor  7  (Information  Search  Activity).  Beta  weights 
and  significance  levels  are  again  presented  in  Table  4. 

The  results  point  toward  strategic  competence  (especially  at 
high  levels),  to  the  strategic  application  of  search  activity, 
toward  appropriate  response  frequency  and  speed  and  toward  the 
avoidance  of  simplistic  responding  to  information  as  likely 
antecedents  of  managerial  success. 

Secondary  Analysis 

A  second  stepwise  regression  analysis  repeated  the  procedure 
above.  However,  in  advance  of  this  regression  procedure,  scores 
obtained  from  the  Shamba  simulation  and  those  obtained  in  the 
Disaster  simulation  were  matched  on  the  basis  of  earlier 
responses  by  previous  simulation  participants.  The  latter  had 
(in  random  order)  performed  in  both  simulations.  The  obtained 
regression  results  did  not  differ  widely  from  the  primary 
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regression  data  presented  in  Table  4: 

1.  Regression  of  Matched  Simulation  Measures  on  Job  Level 

Only  Measure  21P  (Integration  Across  Categories  as  a 
Proportion  of  Number  of  Decisions)  predicted  the  attained  job 
level  of  respondents.  Beta  weights  and  significance  levels  are 
presented  in  Table  5. 

2.  Regression  of  Matched  Simulation  Measures  on  Personal  Income 

Personal  Income  was  significantly  predicted  by  Measure  18 
(Most  Recent  Response  Speed)  and  Measure  28P  (Integrated 
Information  Search  Decisions  as  a  Proportion  of  the  Number  of 
Decisions).  These  data  are  highly  similar  to  the  results 
obtained  earlier  for  unmatched  simulation  measures.  The 
resulting  beta  weights  and  significance  levels  are  again  found  in 
Table  5 . 

3.  Regression  of  Factor  Scores  from  Matched  Data  on  Job  Level 

Attained  Job  Level  was  predicted  by  Factor  4  (Complex 
Strategy)  and  Factor  11  (Recovery).  Beta  weights  and  p  values 
are  found  in  Table  5. 

4.  Regression  of  Factor  Scores  from  Matched  Data  on  Personal 

Income 

Personal  Income  was  predicted  by  Factor  9  (Response  Speed) 
and  Factor  7  (Information  Search  Activity).  The  obtained 
predictors  again  match  those  generated  by  factors  derived  from 
unmatched  simulation  data.  The  resulting  beta  weights  and 
significance  levels  are  again  found  in  Table  5. 


TABLE  5 


Prediction 
Job  Level 

Personnel 

Income 


Prediction 
Job  Level 

Personal 

Income 


STEPWISE  REGRESSION  BASED  UPON 
TRANSFORMED  SIMULATION  DATA 


of  Measure  Beta  Weight 

21P  Proportion  of 

Integrations  across 
Categories  .  29 


18  Most  Recent 

Response  Speed 

28P  Information  Search 
Decisions  Integrated 


.30 

.23 


of 


Factor 

4  Complex  Strategy 
11  Recovery 

9  Response  Speed 
7  Information  Search 


Beta  Weight 
.56 
-.40 


.30 


SUMMARY 


It  is  clearly  evident  that  a  number  of  simulation  based 
measures  as  well  as  some  of  the  factor  scores  are  predictive  of 
managerial  success.  Such  findings  should  be  expected  on  the 
basis  of  complexity  theory  (Streufert  and  Streufert,  1978; 
Streufert  and  Swezey,  1986),  especially  where  participants  in 
simulations  are  senior  managers.  It  should,  however,  be  noted 
that  the  participants  in  this  research  were  drawn  from  a 
population  with  generally  lower  levels  of  managerial  experience. 
At  those  lower  levels,  however,  the  impact  of  education, 
training,  and  previous  relevant  experience,  etc.,  upon 
performance  and  success  would  likely  be  considerably  greater,  in 
turn  diminishing  the  impact  of  simulation  based  measurement  upon 
the  prediction  of  achieved  success.  Since,  nonetheless, 
considerable  (significant)  predictive  capacity  was  obtained, 
simulation  measurement,  especially  where  it  can  be  based  on 
multiple  (independent)  scoring  of  several  different  performance 
attributes,  can  be  very  useful.  While  the  present  data  suggest 
quite  adequate  validity  levels,  much  higher  levels  of  predictive 
validity  should  be  expected  at  more  senior  managerial  levels, 
i.e.,  wherever  the  impact  of  training,  experience,  etc.,  is 
diminished  in  importance. 

It  may  be  useful  to  summarize  the  obtained  results  in  more 
popular  terminology.  the  following  list  of  managerial 
performance  characteristics  which  are  assessed  by  scores  obtained 
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via  our  simulations*  have  been  shown  to  be  significant  predictors 
of  managerial  success: 

1.  Adequate  attention  to  problems/emergencies  (Measure 
33). 

2.  Appropriate  Information  Search  Activity  and  Information 
Utilization  (Measures  27P,  28P  and  Factor  7). 

3.  Time  Length  of  Planning  (Measure  6Q  -  negative  impact). 

4.  Delay  between  Information  Receipt  and  Response 
(Thinking  Time  before  Initiating  Action:  Measures  11, 
18,  29  and  Factor  9). 

5.  Simplistic  Actions  (negative  impact,  Measures  8,  8P). 

6.  Breadth  of  Approach  to  Complex  Task  Demands  (Measure 
2  1 P )  . 

7.  Applications  of  Basic  Strategy  (Measure  4P). 

8.  Quality  of  Strategy  (including  higher  level  strategies, 
Measures  9,  9Q  and  Factor  4). 

9.  The  Capacity  to  Shift  from  a  Strategic  to  a  Problem 
Oriented  Approach,  as  Needed,  and  the  Capacity  to 
Return  to  Strategic  Thinking  when  Emergencies  are 
Resolved  or  Problems  are  Diminished  (Measure  26  and 
Factor  11). 


The  Measures  or  Factors  that  are  associated  with  each  of 
the  more  popular  performance  descriptions  will  be  listed  in 
parentheses . 
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